AdaTask: A Task-Aware Adaptive Learning Rate Approach to Multi-Task Learning

Multi-task learning (MTL) models have demonstrated impressive results in computer vision, natural language processing, and recommender systems. Even though many approaches been proposed, how well these balance different tasks on each parameter still remains unclear. In this paper, we propose to measure the task dominance degree of a by total updates parameter. Specifically, compute exponentially decaying Average squared Updates (AU) from corresponding task. Based novel metric, observe that parameters existing MTL methods, especially those higher shared layers, are dominated one or several tasks. The AU is mainly due accumulative gradients Motivated this, Task-wise Adaptive rate approach, AdaTask short, separate hence for adaptive (e.g., AdaGrad, RMSProp, Adam). Comprehensive experiments vision system datasets demonstrate significantly improves performance tasks, resulting SOTA average task-wise performance. Analysis both synthetic real-world shows every layer well.